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THE  INEFFICIENCY  OF  LEAST  SQUARES: 
EXTENSIONS  OF  KANTOROVICH  INEQUALITY 


C.  Radhakrishna  Rao 


ABSTRACT 

Four  different  measures  of  inefficiency  of  the  simple  least 
squares  estimator  in  the  general  Gauss-Markoff  model  are  considered. 
Previous  work  on  the  bounds  to  some  of  these  measures  is  briefly 
reviewed  and  new  bounds  are  obtained  for  a  particular  measure.  ( 

Keywords:  Inefficiency  of  least  squares,  Kantorovich  inequality. 
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1 .  INTRODUCTION 


Let  us  consider  the  usual  Gauss-Markof f  model 

Y  =  XB  +  e,  E(e )  =  0,  D(e)  =  a2V  (1.1) 


where  Y  and  e  are  n-vectors,  B  is  an  m-vector,  X  is  an  n *  m  matrix  and 
2 

o  V  ,  the  dispersion  matrix  of  e ,  is  positive  definite.  In  practice 
V  may  be  unknown  in  which  case  the  estimate  of  B  is  computed  by 
choosing  an  apriori  dispersion  matrix  in  the  place  of  V.  A  number 
of  authors  have  investigated  the  loss  of  information  in  the  estimation 
of  S  resulting  from  a  wrong  choice  of  V.  See  for  instance,  Bloomfield 
and  Watson  (1975),  Khatri  and  Rao  (1981,1982),  Knott  (1975)  and 
Styan  (1983).  The  object  of  the  present  paper  is  to  review  some  of  the 
earlier  results  and  provide  a  generalization  of  a  recent  result  by 
Styan  (1983) . 


There  is  no  loss  of  generality  in  assuming  V  =  I, for  the  problem 

3 

associated  with  V  and  V,  when  V  is  positive  definite,  is  the  same  as 

3  3 

-1/2  -1/2 

that  with  I  and  V  V  V  for  purposes  of  the  present  investigation. 
3  3 

Also,  we  consider  the  basic  parameter  as  XB  and  study  the  inefficiency 
of  its  estimation  due  to  a  wrong  choice  of  V.  In  such  a  case,  we  can, 
without  Loss  of  generality,  consider  X  to  be  of  full  rank  with  its 
column  vectors  as  orthonormal.  Then  the  simple  least  squares  estimator 
and  the  BLUE  of  XB  are 


XX 'Y  and  X(X’V_1X)“1X,V-1Y  (1.2) 

2 

with  the  dispersion  matrices  (apart  from  the  multiplying  constant  a  ) 

XX'VXX'  and  X(X'v'1X)"1X'  . 


(1.3) 


If  V  =  P  P'  is  the  spectral  decomposition  of  V,  where  P  is  an  ortho¬ 
gonal  matrix  and  A  is  a  diagonal  matrix  of  eigenvalues  of  V,  then 
the  matrices  in  (1.3)  can  be  written  as 

PUU*  A  UU’P*  and  PU(U*A-1U) '1U 'P ’  (1.4) 

where  U  =  P'X  and  hence  U'U  =  I.  In  the  next  section  we  define  a 
number  of  measures  of  inefficiency  based  on  a  comparison  of  the 
matrices  in  (1.3)  for  (1.4)  and  determine  their  lower  and  upper  bounds 
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2.  MEASURES  OF  INEFFICIENCY  AND  THEIR  BOUNDS 


By  construction,  the  difference  between  the  first  and  second 
matrices  in  (1.4)  is  non-negative  definite  and  the  magnitude  of  the 
difference  can  be  judged  by  the  magnitudes  of  the  proper  nonzero 
eigenvalues  (Rao  and  Mitra,  1971,  124-126)  of  the  first  matrix  with 
respect  to  the  second.  If  9  is  a  proper  nonzero  eigenvalue,  then 


puu' a  uu ' p * x  =  epu(u'A-1u)-1u'p'x 


(2.1) 


where  the  left  and  right  hand  sides  of  (2.1)  do  not  individually 
vanish.  In  such  a  case,  multiplying  both  sides  of  (2.1)  by  U'P'  and 
writing  y  =  U'P'x,  we  have 


U'  A  Uy  =  6(U,A-1U)“1y 


(2.2) 


so  that  9  is  a  root  of  the  determinental  equation 

| U' A  U  -  9(U,A_1U)"1|  =  0. 


(2.3) 


We  may  choose  any  increasing  function  of  the  roots  of 

(2.3)  as  a  measure  of  inefficiency,  such  as  9.  ...  9  or  9  +  ...  +  0  . 

i  m  i  m 

Bloomfield  and  Watson  (1975)  and  Knott  (1975)  have  established  the 
bounds 


m  s  (A  .  + A  .  . )‘ 

,  _  „  v  i  n-i+l 

i  <  n  e,  <  I 


1  1  i'1  4iiX„-l+l 


(2.4) 


where  A,  >  A_  ...  >  A  are  the  diagonal  elements  of  A  (i.e.,  the 
eigenvalues  of  V)  and  s  =  min(m,  n-m) .  Khatri  and  Rao  (1981)  estab¬ 
lished  that 

m 


m 


<  l  e  <  l  (Ai  +  An~i--  +  t 
1  1  4Vn-l+l 


(2.5) 


where  s  =  min(m>  n-m),  and  t  ■  0  if  s  *  m  and  t  =  2m -  n  if  s  =  n-m. 


Puntanen  (1982)  suggested  the  use  of 


i 

s 

i 

t 

i 


I 


« 


« 


« 


tr (PUU ' A  UU'P '  -  PU(U'A_1U)“1U'P')  (2.6) 

as  a  measure  of  inefficiency.  The  expression  (2.6)  reduces  to 

tr(U'  A  U  -  (U'A-1U)-1).  (2.7) 

In  the  special  case  when  m  =  1,  Styan  (1983)  showed  that 

0  <  U'AU  -  (U'A-1U)-1  <  (AT  -  A~“  )2.  (2.8) 

—  in 

We  provide  the  following  generalization  for  higher  values  of  m. 

Theorem .  For  general  m 

g 

0  <  tr(U’AU  -  (U'A^U)-"1)  <  f  (AT  -  A  “7  )2  (2.9) 

—  —  i“i  l  n-i+l 

where  s  =  min(m  f  n-m) . 

Proof .  We  find  the  stationary  values  of  (2.7)  subject  to  the 
condition  U'U  =  I.  Introducing  a  symmetric  matrix  A  of  Lagrangian 
multipliers  we  consider  the  expression 

tr  U'AU  -  tr (U 'A-1U)_1  -  tr  A(U'D-I).  (2.10) 

Taking  derivatives  of  (2.10)  with  respect  to  the  elements  of  U  (see 
Rao,  1984)  and  equating  to  zero,  we  have 

U'A  +  (U'A-1U)-2  U'A-1  =  AU'  (2.11) 

which  gives 

U'  A  U  +  (U'A_1U) _1  =  A.  (2.12) 

Then  the  equation  (2.11)  reduces  to 

U'A  +  (U’A_1U)~2  U'A-1  =  (U'AU  +  (U ' A_1U) -1) U ’ .  (2.13) 

Multiplying  both  sides  of  (2.13)  from  the  right  by  AU 

U'  A2  U  +  (U'A^U)'1  =  (U'AU)2+(U’A'1U)'1(U'AU)  (2.14) 


< 
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which  shows  chat  the  last  term  in  (2.14)  is  symmetric  or  the  matrices 
U'  A  U  and  U'A  commute.  Then,  there  exists  an  orthogonal  matrix  Q 


such  that 


U '  A  U  =  QEQ'  and  U'A1U=  Q  A  Q' 


(2.15) 


where  A  and  E  are  diagonal  matrices  with  diagonal  elements,  say, 
dj_,...,d^  el’‘"’,em‘  Writing  W  =  UQ,  the  equation  (2.13)  becomes 


AW  +  A_1  W  A~2  =  W(E  +  A_1). 


(2.16) 


Let  (w, , . . . ,w  )'  be  the  j-th  column  of  W.  Then 
in 

X.w.  +  X  ^  w.  d.2  =  w.(e.  +  d.^),  i  =  l,...,n 
li  iij  lj  j  * 


(2.17) 


which  shows  that  at  most  two  values  of  w.  can  be  nonzero.  If  w  and 


w  are  non-zero,  then 
s 


-1  -2  -1 
X  +  X  d.  =  (e.  +  d.  ) 
J  J  J 


(2.18) 


has  two  roots  X  and  X  ,  and  it  is  seen  that 
r  s 


e.  -  dT  =  (ST  -  /X~  )‘ 
3  3  r  s 


(2.19) 


If  only  one  w^  is  non-zero,  then 


e.  -  d  71  =  0. 
J  3 


(2.20) 


The  expression  we  have  to  maximize  is 


tr(U' A  U  -  (U 


’A-1U)_1)  =  tr(E  -  A-1) 


=  1  (e.  -  dT1) 


J  3 


(2.21) 


where  each  term  in  (2.21)  has  the  value  zero  as  in  (2.20)  or  a  value  of 
the  type  («^X~  -  /xj  )2  as  in  (2.19).  Using  arguments  similar  to  those 
in  Bloomfield  and  Watson  (1975)  and  Knott  (1975),  we  find  the  maximum 


of  (2.7)  is 


-  ^-i+i  y 


(2.22) 


where  s  -  min(m,  n-m),  which  proves  the  required  result. 


Remark  1.  In  terms  of  the  original  matrices  X  and  V  where  X  need 
not  be  assumed  to  have  orthonormal  columns,  the  matrices  in  (1.3)  can 
be  written  as 


Px  VPX  and  PX(PXV-1  Px)  Px  (2.23) 

where  ?x  =  X(X'X)  X',  the  projection  operator  on  the  space  generated 
by  the  columns  of  X  and  (•)  denotes  any  generalized  inverse.  Then  the 
result  (2.9)  of  the  Theorem  can  be  written  as 


0  < 


tr(psVPx  - 


px<px','lpx>' 


f)  1  l  iA ~  -  A 


i=l 


n-i+1 


(2.24) 


where  X^  are  the  eigenvalues  of  V.  Let  be  the  corresponding  eigen¬ 
vectors  and  denote 


£ 


i 


AT 

i 


A 


n-i+l 


^  +  ^n-i+1 


Xn-i+l 

(2.25) 


It  is  seen  that  the  upper  bound  in  (2.24)  is  attained  when  the  columns 
of  X  are  generated  by  the  vectors  and  some  vectors 

orthogonal  to  £^,...,£  . 

Remark  2 .  When  m  =  1,  simple  proofs  are  available  for  the 
inequalities  (2.4)  and  (2.9)  as  given  by  Styan  (1983). 
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3.  ANOTHER  MEASURE  OF  INEFFICIENCY 


In  a  paper  presented  at  the  Fifth  Berkeley  Symposium  in  1965,  the 
author  showed  that  there  is  no  loss  of  information  in  estimation  by 
simple  least  squares  if  V  and  X  satisfy  the  condition  X'VZ  =  0  where 
Z  is  any  matrix  with  maximum  rank  such  that  X'Z  =  0  (see  Rao,  1967). 
The  equivalent  condition  P  V  =  VP  was  given  by  Zyskind  (1967) .  The 

X.  A 

condition  X'VZ  =  0  =  P  VZ  is  equivalent  to 

A 

0  =  PXVZ(Z'Z)“  Z'VPX  =  P  V(I  -  Px)  VPX 

=  PXV2PX  -  (PXVPX)(PXVPX)  .  (3.1) 

Bloomfield  and  Watson  (1975)  considered  (3.1)  as  a  measure  of 


4 .  REFERENCES 


[1]  BLOOMFIELD,  P.  and  WATSON,  C.S.  (1975).  The  inefficiency  of 
least  squares.  Biometrika  62,  121-128. 

[2]  KHATRI ,  C.G.  and  RAO,  C.R.  (1981).  Some  extensions  of  the 
Kantorovich  inequality  and  statistical  applications.  J.  Multi¬ 
variate  Analysis  11,  498-505. 

[3]  KHATRI,  C.G.  and  RAO,  C.R.  (1982).  Some  generalizations  of 
Kantorovich  inequality.  Sankhya  A  44,  91-102. 

[4]  KNOTT,  M.  (1975).  On  the  minimum  efficiency  of  least  squares. 
Biometrika  62,  129-132. 

[5]  PUNTANEN  SIMO  (1982).  Personal  communication. 

[6]  RAO,  C.R.  (1967).  Least  squares  theory  using  an  estimated  dis¬ 
persion  matrix  and  its  application  to  measurement  of  signals. 

Proc.  Fifth  Berkeley  Symp.  355-373. 

[7]  RAO,  C.R.  and  MITRA,  S.K.  (1971).  Ge*  , ized  Inverse  of  Matrices 
and  its  Applications,  Wiley,  New  York 

[8]  RAO,  C.  RADHAKRISHNA  (1984).  Matrix  der.  rves :  Applications  in 
Statistics.  Encyclopedia  of  Statistical  Sciences,  (Eds.  S.  Kotz 
and  N.  L.  Johnson),  Wiley. 

[9]  STYAN,  G.P.H.  (1983).  On  some  inequalities  associated  with 
ordinary  least  squares  and  the  Kantorovich  inequality.  In 
Festschrift  for  Eino  Haikala  on  his  Seventieth  Birthday, 

University  of  Tampere,  158-166. 


_ UNCLASSIFIED  _ 

Sf  Cw<*  TV  C.ASS.F'CAT.ON  OF  THIS  PAGE 


r ' 


1*  fi£PC«T  SEC^R'TV  C^ASSi^CATON 

UNCLASSIFIED 


?fc  StCuPiT>  CLASSIFICATION  AuTmQRiTY 


2b  DEC  L  ASS  <  F  *  C  A  T  •  QN  DOWNGRADING  SChE  DUL6 


A  *fc«FO*M.NG  ORGANi2ATiCN  REPORT  NcMBERlS) 

84-47 


REPORT  DOCUMENTATION  PAGE 


ID  RESTRICTIVE  MARKINGS 


lutnutMaa 


3  distribution. a  vailab'lity  of  report 
Approved  for  public  release;  distribution 
unlimited . 


5  MONITORING  ORGANIZATION  REPORT  NUMBERlSl 

XFOSR-TR.  24.1199 


6a  NAME  op  PERFORMING  ORGANIZATION  6b.  OFFICE  SYMBOL  7a  NAME  Of  MONITORING  ORGANIZATION 

(If  apphcabU) 

Universitv  of  Pittsburgh  ,  .  _  n{t,  ,  _  ,  _.  . 

Air  Force  Office  of  Scientific  Research 


6c.  AOORESS  'Cif>  Slot*  ana  ZIP  CMlt I 

Center  for  Multivariate  Analysis 
Fifth  Floor,  Thackeray  Hall 
Pittsburgh  PA  15260 

a.  na»jf  nr  f  L,Nn.Nr.  er.ksra.Lr,  Yj,., 


7b  ADDRESS  (City.  Statt  ana  ZIP  CaOtl 

Directorate  of  Mathematical  &  Information 
Sciences,  Bolling  AFB  DC  20332-6A4S 


6a  NAME  OF  F  OND.NG  SPONSOR. NG 
ORGANIZATION 

AFOSR 


Sc  ADDRESS  'City.  Slale  and  ZIP  Code/ 


ftt>  OFFICE  SYMBOL  9  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 
{If  applicable/ 

MM  F49620-82-K-0001 


10  SOURCE  OF  FUNDING  NOS 


PROGRAM  PROJECT 

E  LEMENT  NO.  NO. 


WORK  UNIT 
NO 


Bolling  AFB  DC  20332-6448  61102F  2304 


11  TITLE  'Include  Security  Clout ficotioni 

THE  INEFFICIENCY  OF  LEAST  SQUARES:  EXTENSIONS  OF  KANTOROVICH  INEQUALITY 


12  PERSONAL  AUThORISI 

C.  Radh.akrishna 


PQ 


7 

3c  "*  ME  COVE  RE 

FROM 

TO _ 

14  DAT  EOF  REPORT  >  Yr  .  Mo  .  Day/ 

\'0V  8- 

15  PAGE  COUNT 

8 

COSAT*  CODES 


FiElD  I  GROUP 


1ft  SUBJECT  TERMS  (Continue  on  reue^te  if  neceuory  and  identify  by  block  number) 

Inefficiency  of  least  squares;  Kantorovich  inequality. 


1ft  ABSTRACT  { Continue  on  *» if  nectieary  and  identify  by  Noe*  number) 

Four  different  measures  of  inefficiency  of  the  simple  least  squares  estimator  in  the  genera] 
Gauss -Markov  model  are  considered.  Previous  work  on  the  bounds  to  some  of  these  measures 
is  brieflv  reviewed  and  new  bounds  are  obtained  for  a  particular  measure. 


[20  DISTRIBUTION. 'AVAILABILITY  OF  ABSTRACT 


2V  ABSTRACT  SECURITY  CLASSIFICATION 


UN  C  LASS  I  FlEO,'UNLlMlTED  E  SAME  AS  RPT  C  DTlC  USERS  □  UNCLASSIFIED  ^  01  16  104 

122%.  NAME  OF  RESPONSIBLE  INDIVIDUAL  22b  TELEPHONE  NUMBER  22c  OFFICE  SYMBOL 

(Include  A>-*a  Code ) 

MAJ  Brian  W.  Woodruff  (202)  767-  5027  nm 

DO  FORM  1473,83  APR 


22b  TELEPHONE  NUMBER 
(Include  A  Yfl  Code > 

(702^  767-  5027 


:  ICE  SYMBOL 


(202)  767 

MMIWMWWWWW.1W  Aww  —p 

EDITION  OF  1  JAN  73  IS  OBSOLETE. 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  THIS  PA< 


FILMED 

1 

5  2-85 

< 

DTIC 


